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AGENDA 




1. Evaluation Policy 

History of Evaluation Policy in the U.S. 
Obama Administration Policy 
ILAB Evaluation Policy 

2. Case Studies: ILAB Africa Projects 

3. Challenges in Meeting Policy Goals 

Problem Definition 
Recommendations 

4. Recommendations 

5. Questions/Discussion 



SHIFTS IN USG EVALUATION POLICY VIS-A-VIS FOREIGN 

ASSISTANCE (1993-2008) 




1993: Government Performance and Results Act (GPRA) 



o Shift towards results-oriented measurement of federal 
programs 

2003: Program Assessment Rating Tool (PART) 

o Focus on encouraging agencies to use the type of evaluation 
that will best demonstrate the effectiveness of their programs 

2005: Secretary of State Condoleezza Rice calls for foreign- 
assistance strategy that includes: 

"a system of coordinated planning, budgeting, and evaluation." 




SHIFTS IN USG EVALUATION POLICY VIS-A-VIS FOREIGN 

ASSISTANCE (2009 - PRESENT) 




July 10, 2009: 

Hillary Clinton calls for a 
quadrennial diplomacy and 
development review (QDDR) 



Recent Legislation 



August 31, 2009: 

President Obama announces a 
Presidential Study Directive on 
Global Development Policy 

-Program sustainability 
-Larger investment in M&E 



April 28, 2009: H.R. 2139, Initiating 
Foreign Assistance Reform Act of 
2009 introduced 

May 13, 2009: H.R. 2387, Strategy 
and Effectiveness of Foreign Policy 
and Assistance Act of 2009 
introduced 

July 28, 2009: S. 1524, Foreign 
Assistance Revitalization and 
Accountability Act of 2009 
introduced 



Impact Evaluations 



Steps Taken to Promote 
Impact Evaluations 



The Obama Administration is 
focusing its attention on 
impact evaluations over 
process evaluations 



Plan to post all USG impact 
evaluations online 

Interagency working group 

Voluntary evaluation initiative 



CURRENT USG EVALUATION POLICY CONT. 




Other Administration Efforts: 



Transparency of evaluations 

o Effort to make evaluations available to other USG agencies, 
researchers, and public 



Whole-of-government approach to foreign aid 

o Effort to coordinate the activities of all development-focused 
agencies 



CURRENT DOL/ILAB EVALUATION POLICY 

If ^ 








Current USG 
Evaluation Policy 


International Labor Affairs Bureau 
OCFT OTLA 


Method 


Impact evaluations 


Process evaluations, 
moving toward impact 


Process evaluations, 
moving toward impact 


Transparency 


Transparency of 
evaluation process 
important 


Evaluation summaries 
posted on the web 


Evaluations only 
available within the 
Bureau; reports 
available upon request 


Uses 


(1) Determine cost 
effectiveness 

(2) Strengthen design 
and operation of 
programs 

(3) Shape budget 
priorities 


(1) Assess if desired 
outcomes reached 

(2) Improve program 
design and 
management 

(3) Achieve cost 
efficiencies 

(4) Program sustainability 


(1) Enhance 
management of 
ongoing projects 

(2) Improve preparation 
of new projects 

(3) Provide inputs into 
broader program 
evaluations 





DOL/ILAB CASE STUDIES 





CASE STUDIES OF DOL-FUNDED LABOR 
PROJECTS IN AFRICA WILL BE DISCUSSED AS 
THEY RELATE TO 2 QUESTIONS: 

l) Did evaluations foster program improvements? 




2) How transparent was the evaluative process? 



PROJECTS USED AS CASE STUDIES 

(Did evaluations foster program improvements?) 




1) STRENGTHENING LABOR ADMINISTRATION IN SOUTHERN 
AFRICA (SLASA) 

Midterm Evaluation: October 31, 2003 

2) IMPROVING INDUSTRIAL RELATIONS IN MOZAMBIQUE 

o Final Report by USDOL: December 11, 2003 

3) SOUTH AFRICAN VETERANS EMPLOYMENT PROGRAM 
(TSWELOPELE) 

Final Evaluation: December 10, 2004 



4) NIGERIAN VETERANS EMPLOYMENT PROGRAM 

Final Evaluation: October 1, 2004 



PROJECTS USED AS CASE STUDIES CONT. 

(Did evaluations foster program improvements?) 




5) TANZANIA LABOR EXCHANGE CENTER (LEC) PROJECT 

Final Evaluation: February 2004 



6) NIGERIAN DECLARATION (NIDEC) PROJECT 

Final Evaluation: June 24, 2005 



7) IMPROVING LABOR SYSTEMS IN SOUTHERN AFRICA (ILSSA) 

Midterm Evaluation: October 2006 



i) Developing Effective 
Indicators for 
Measuring Progress 



o 




Six of the seven projects 
reviewed experienced serious 
problems with their 
performance monitoring plans 
(PMPs), leading to a lack of 
information about the projects' 
outcomes. 



Common Problems Found in 
Evaluations 




(1) Lack of timeliness in PMP 
design 

(2) Poor drafting of indicators 

(3) Incompletion or 
unsatisfactoriness of PMP 

(4) Reporting of indicator data not 
useful 



RECOMMENDATIONS for INTERNAL PROGRAM 

IMPROVEMENT 



Conducting Process 
Evaluations 




Impact versus Process 
Evaluations 



Agencies should: 

(a) Focus on accuracy of 
indicators and quality of 
reporting 

(b) Budget an appropriate 
amount of time for the design 
of a PMP with effective 
progress-measuring indicators 



Agencies should: 

(a) Balance and prioritize impact 
and process evaluations based 
on what needs to be learned 
about the project 

(b) Select and tailor 
methodologies accordingly 



i) Fostering Institutional 
Learning 



o 



Common problems in project 
design and implementation 
were found in the evaluations 
reviewed, suggesting a lack of 
institutional learning that 
would have reduced the 
severity of such problems over 
time. 



Common Problems Found in 
Evaluations 




TRANSPARENCY CASE STUDY 
(How transparent is the evaluative process?) 




4 Key Documents Reviewed 




OCFT Procedures for Working with Contractor- Led Evaluations 
TOR for ILO-IPEC Mali Final Evaluation (OCFT May-June 2010) 
TAATC Guide to Evaluations 

TOR for ILSSA Final Evaluation (TAATC July 2008) 



2) Focusing on 
Transparency 



o 




There is currently a lack of 
transparency in the evaluation 
process, further limiting 
knowledge transfer achieved 
through the evaluation process. 



Common Problems 

(1) Key agency evaluation 
documents lack reference to 
transparency. 

(2) Key project agreement 
documents lack reference to 
transparency of evaluations. 



(3) Staff performance measures 
lack reference to transparency 
of evaluation processes or 
outputs. 



Transferring Knowledge 



Increasing Transparency 



Agencies should: 

(a) Make process and impact 
evaluations available within 
the agency and, preferably, to 
other USG agencies 

(b) Summarize the findings of 
evaluations completed within 
the agency and keep track of 
common trends 



Agencies should: 

(a) Make better use of the Web to 
post project evaluation plans, 
requests for stakeholder 
input, scheduled evaluation 
missions, and evaluators' 
reports for broader USG and 
public access 



CONCLUSION 




ILAB's Africa projects over the past decade produce a 
compelling example of what needs to be done at the 
agency level to improve the transparency of the 
evaluation process and the knowledge transfer 
necessary to improve foreign-assistance projects. 
Separate from Administration policy, agencies need to 
take concrete steps to weave evaluation measures into 
project design as well as to improve their mechanisms 
for transferring the knowledge gained from evaluations 
as a way to increase institutional learning. 



SMALL GROUP BRAINSTORMING 




1. Integrating evaluation planning in program design 

2. Transitioning to impact-oriented evaluation 
methodologies 

3. Enhancing evaluation processes to facilitate knowledge 
transfer 

4. Increasing transparency of evaluations and evaluation 
processes 

* Depending on the # of participants, Groups 1&2 and Groups 3&4 may be 
combined. 




